Measuring the quality of multi-document cluster headlines

Headline summaries of multi-document clusters enable efficient navigation and selection of content, provided headlines are of sufficient quality. This study compares several methods for automated headline extraction, with considerable variation in length. The reliability of the automated evaluation is validated by a comparison with human produced headlines, taking into consideration the variability in manually created headlines and inter-human agreement in quality judgements. Results suggest that, ROUGE precision is a suitable measure for automatic evaluation of headlines of differing lengths. Also ROUGE Recall can be used after applying a length penalty.